Repository navigation
fix(writer): the child nodes of a flat segment carry the array stats Rust keeps (#583) - #674
Merged
Merged
Conversation
…Rust keeps (#583) Rust's compressor leaves its cached stats on every array it compresses, so the children a scheme cascades into carry them as the root does. The cascade now attaches the same stats to each child it encodes (RootNodeStats moves to writer.encode), and three shapes of their own: - vortex.sequence writes is_sorted / is_strict_sorted (Sequence::stats: false for a falling step, true for fewer than two values) - the primitive values of ALP and ALP-RD patches carry is_constant = false - the vortex.bool validity bitmap of a nullable primitive is is_constant when every row has one value, otherwise min, max, sum (valid rows) and null_count = 0; NodeStats gains the sum component NodeStatsInteropIntegrationTest now compares the whole tree of the numeric columns against vortex-jni, with patched-float and falling-sequence columns. String children (FSST/onpair, which depend on the trained symbol table) and the zone-table segments of string and bool columns remain. Refs #583 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
dfa1
added a commit
that referenced
this pull request
Nov 6, 2026
… with unit tests (#583) Review of #674: RootNodeStats is now the package-private CachedNodeStats, reached from VortexWriter through one public EncodeResult#withRootStats. Unit tests cover the cached stats on dict children, the validity bitmaps of a nullable integer and float, patch values and the fill-only-when-empty rule; the interop test gains a nullable f64 column. Docs: three shapes, and bit-packed patch values. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of #583 (after #673).
The cascade attaches Rust's stats to every child it encodes, not just the root, plus sequence sorted flags, patch-values
is_constant=false, and the validity bitmap stats (is_constant, ormin/max/sum/null_count).NodeStatsInteropIntegrationTestcompares the whole tree of the numeric columns against vortex-jni.Not covered yet: string children (FSST/onpair values depend on the symbol table) and the zone-table segment shape of string and bool columns (PR 3).
Refs #583
🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV